Device and Method of Handling Multiple Formats of a Video Sequence

ABSTRACT

A transmitter for handling multiple formats of a video sequence, comprises a preprocessing module, for receiving a first format of a video sequence, to generate metadata of a second format of the video sequence according to the first format of the video sequence and the second format of the video sequence; and an encoder, couple to the preprocessing module, for transmitting the first format of the video sequence and the metadata in a bit stream to a receiver.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/579,902 filed on Nov. 1, 2017, which is incorporated herein byreference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a device and a method used in amultimedia communication system, and more particularly, to a device anda method of handling multiple formats of a video sequence.

2. Description of the Prior Art

Due to the richness and development of multimedia services, great amountof multimedia content is transmitted via communication networks. Forexample, a transmitter may transmit video sequences to a receiver, whilethe video sequences are generated according to a same video source.Simply transmitting the video sequences results great consumption ofbandwidth and power. Thus, transmitting the video sequences efficientlyis an important problem to be solved.

SUMMARY OF THE INVENTION

The present invention therefore provides a method and relatedcommunication device for handling multiple formats of a video sequenceto solve the abovementioned problem.

A transmitter for handling multiple formats of a video sequence,comprises a preprocessing module, for receiving a first format of avideo sequence, to generate metadata of a second format of the videosequence according to the first format of the video sequence and thesecond format of the video sequence; and an encoder, couple to thepreprocessing module, for transmitting the first format of the videosequence and the metadata in a bit stream to a receiver.

A receiver for handling multiple formats of a video sequence, comprisesa decoder, for receiving a bit stream comprising a first format of avideo sequence and metadata of a second format of the video sequencefrom a transmitter; and a postprocessing module, couple to the decoder,for generating the second format of the video sequence according to themetadata and the first format of the video sequence.

These and other objectives of the present invention will no doubt becomeobvious to those of ordinary skill in the art after reading thefollowing detailed description of the preferred embodiment that isillustrated in the various figures and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an end-to-end system with a base video directly viewablefeature according to an example of the present invention.

FIG. 2 is a transmitter according to an example of the presentinvention.

FIG. 3 is a transmitter according to an example of the presentinvention.

FIG. 4 is a receiver according to an example of the present invention.

FIG. 5 is a transmitter according to an example of the presentinvention.

FIG. 6 is a transmitter according to an example of the presentinvention.

FIG. 7 is a receiver according to an example of the present invention.

FIG. 8 is a transmitter according to an example of the presentinvention.

FIG. 9 is a receiver according to an example of the present invention.

FIG. 10 is a HDR generator according to an example of the presentinvention.

FIG. 11 is a SDR generator according to an example of the presentinvention.

FIG. 12 is a SDR generator according to an example of the presentinvention.

FIG. 13 is a schematic diagram of division of a base video color spaceaccording to an example of the present invention.

DETAILED DESCRIPTION

FIG. 1 is an end-to-end system 10 with a base video directly viewablefeature according to an example of the present invention. There are atransmitter TX and a receiver RX in the end-to-end system 10. In detail,the transmitter TX includes a base video sequence 100, a preprocessingmodule 110, additional format(s) of the base video sequence 112 andmetadata 120 and an encoder 130. The receiver RX includes a decoder 140,a base video sequence 150, metadata 152, a postprocessing module 160 andadditional format (s) of the base video sequence 170. The preprocessingmodule 110 receives the base video sequence 100, and processes the basevideo sequence 100 and the additional format(s) of the base videosequence 112 to generate the metadata 120. The metadata 120 includesadditional information (i.e., supplementary information) for generatingthe additional format(s) of the base video sequence 112 according to thebase video sequence 100 and the metadata 120. Then, the transmitter TXcompresses and transmits the base video sequence 100 and the metadata120 in a bit stream 180 to the receiver RX.

The decoder 140 decodes the bit stream 180 to generate the base videosequence 150 and the metadata 152, after the receiver RX receives thebit stream 180. The receiver RX may simply playback (or play) the basevideo sequence 150 (e.g., in a regular fashion), if the base videosequence 150 is requested/subscribed. The postprocessing module 160generates the additional format(s) of the base video sequence 170according to the base video sequence 150 and the metadata 152, if theadditional format(s) of the base video sequence 170 isrequested/subscribed. That is, the postprocessing module 160 maps thebase video sequence 150 to the additional format (s) of the base videosequence 170 according to the metadata 152. Then, the receiver RXplaybacks (or plays) the additional format(s) of the base video sequence170.

It should be noted that the base video sequence 150 may be (e.g.,slightly) different from the base video sequence 100, e.g., due to lossycompression or an imperfect channel experienced by the bit stream 180.The metadata 152 may be (e.g., slightly) different from the metadata120, e.g., due to an imperfect channel experienced by the bit stream180. That is, the base video sequence 150 and the metadata 152 may notbe reconstructed perfectly. In one example, the base video sequence 150and the metadata 152 may be the same as the base video sequence 100 andthe metadata 120, respectively.

As can be seen, the present invention provides a single layermulti-application compatible solution for transmitting two or moreformats of a base video sequence. Since the metadata 152 providesrelation(s) between the base video sequence 150 and the additionalformat (s) of the base video sequence 170, a transmission of redundantvideo information is avoided. Thus, consumption of a bandwidth and apower of the transmitter TX can be reduced.

Depending on applications, the base video sequence 100 and theadditional format(s) of the base video sequence 170 can be but notlimited to one of the following combinations: standard dynamicrange/high dynamic range (SDR/HDR), HDR/SDR, or HDR of differentformats.

It should be noted that the bit stream 180 may be seamlessly handled bythe decoder 140 in the art. Thus, there is no need to make anymodification on an existing system to decode and playback receivedcontents. When the additional format(s) of the base video sequence 170is demanded, the post processing module 160 processes the base videosequence 150 and the metadata 152 to reconstruct the additional format(s) of the base video sequence 170. Thus, the one-layer approach is notonly capable of supporting multiple formats of a video sequence, butalso compatible with legacy devices.

The present invention provides a mapping mechanism for transmitting thebase video sequence 100 and the additional format (s) of the base videosequence 112 in the bit stream 180. The mapping mechanism maps the basevideo sequence 100 to the additional format (s) of the base videosequence 112 (e.g., additional supported formats of video sequence ofthe same content), and generates the metadata 120 for the transmittedsequence. The metadata 120 describes a relation between the base videosequence 100 and the additional format(s) of the base video sequence112, and is used in the decoder 140 for reconstructing the additionalformat(s) of the base video sequence 170.

The present invention at least includes the following advantages overthe prior art:

(1) A new constant luminance approach to map HDR to SDR. Instead ofperforming a HDR to SDR conversion in R, G, B components individually,the present invention obtains (e.g., calculates) a constant luminancedirectly, maps a constant luminance to SDR, and uses the constantluminance to decide a value of chromaticity.

(2) A 3D lookup table is used for generating metadata for mapping a basevideo sequence to additional format(s) of the base video sequence.

(3) Although luminance and chromaticity are derived separately, it isproved that these parameters can be derived together in a RGB domain andthus a white point and three-color primaries are naturally maintainedafter conversion.

(4) A complete SDR generation process is performed in a ¼ resolutiondomain, and computational complexity is reduced significantly.

(5) The present invention can realize at least the following threeapplications: the encoder 130 transmitting SDR content and metadata ofHDR content, and the decoder 140 and the postprocessing module 160generating the SDR content and the HDR content; the encoder 130transmitting the HDR content and metadata of the SDR content and thedecoder 140 and the postprocessing module 160 generating the HDR contentand the SDR content; and the encoder 130 transmitting the HDR contentand metadata of another format of the HDR content and the decoder 140and the postprocessing module 160 generating two formats of the HDRcontent. In the following sections, these applications are illustrated.

The first application includes transmitting a SDR video sequence as abase video sequence and metadata of a HDR video sequence. Nowadays, mostTVs can play only SDR content, while emerging HDR TVs are demanding foreye-catching HDR content. In this application, the encoder 130 transmitsthe SDR video sequence and transmits HDR related information in themetadata. At the decoder 140, the SDR video sequence is decoded andplayed by using a current SDR receiver without any modification on theSDR receiver. If a HDR playback is supported/requested, the HDR videosequence can be reconstructed by using the SDR video sequence and themetadata.

FIG. 2 is a transmitter TX1 according to an example of the presentinvention. The transmitter TX1 may be used for realizing the transmitterTX (the preprocessing module 110 and/or the encoder 130) in FIG. 1, andincludes modules performing different signal processing functions on aHDR signal 200 (e.g., RGB signal). The HDR signal 200 may be a HDRsource with a transform curve, and may be generated by a video capturingdevice or an editing device. The HDR signal 200 is processed by using aninverse transform curve function 210, and a resulted signal 210 a (e.g.,linear signal) is generated and provided to a SDR generator 220 and aHDR generator 230. The SDR generator 220 processes the resulted signal210 a, to generate a SDR YUV signal 220 a. The HDR generator 230includes a new transform curve function 232, a color space transformfunction 234 and a downsampling function 236, and processes the resultedsignal 210 a, to generate a HDR YUV signal 230 a (e.g., format 4:2:0). A3D lookup table 240 is used for generating metadata 240 a according tothe SDR YUV signal 220 a and the HDR YUV signal 230 a. The metadata 240a provides additional information by describing a relation between theSDR YUV signal 220 a and the HDR YUV signal 230 a. Then, the transmitterTX1 may transmit the SDR YUV signal 220 a and the metadata 240 a in abit stream, e.g., to the receiver RX.

FIG. 3 is a transmitter TX2 according to an example of the presentinvention. The transmitter TX2 may be used for realizing the transmitterTX (the preprocessing module 110 and/or the encoder 130) in FIG. 1, andincludes modules performing different signal processing functions on aHDR YUV signal 300. A SDR generator 310 processes the HDR YUV signal300, to generate a SDR YUV signal 310 a. A 3D lookup table 320 is usedfor generating metadata 320 a according to the SDR YUV signal 310 a andthe HDR YUV signal 300. The metadata 320 a provides additionalinformation by describing a relation between the SDR YUV signal 310 aand the HDR YUV signal 300. Then, the transmitter TX2 may transmit theSDR YUV signal 310 a and the metadata 320 a in a bit stream, e.g., tothe receiver RX.

FIG. 4 is a receiver RX1 according to an example of the presentinvention. The receiver RX1 may be used for realizing the receiver RX(the decoder 140 and/or the postprocessing module 160) in FIG. 1, andincludes modules performing different signal processing functions on abit stream 400. A video decoder 410 (e.g., audio video coding standard(AVS) decoder) processes the bit stream 400, to generate a SDR YUVsignal 410 a (e.g., format 4:2:0) and metadata 410 b. A 3D lookup table420 is used for generating a HDR YUV signal 420 a (e.g., format 4:2:0)according to the SDR YUV signal 410 a and the metadata 410 b. Then, thereceiver RX1 may playback the HDR YUV signal 420 a.

The second application includes transmitting a HDR video sequence as abase video sequence and metadata of a SDR video sequence. After HDRcontent and HDR players become dominant, most TVs support HDR. As aresult, a requirement for a video transmission may change. In thisapplication, the encoder 130 transmits the HDR video sequence andtransmits SDR related information in the metadata. At the decoder 140,the HDR video sequence is decoded and played by using a current HDRreceiver without any modification on the HDR receiver. If a SDR playbackis needed, the SDR video sequence can be reconstructed by using the HDRvideo sequence and the metadata.

FIG. 5 is a transmitter TX3 according to an example of the presentinvention. The transmitter TX3 may be used for realizing the transmitterTX (the preprocessing module 110 and/or the encoder 130) in FIG. 1, andincludes modules performing different signal processing functions on aHDR signal 500 (e.g., RGB signal). The HDR signal 500 may be a HDRsource with a transform curve, and may be generated by a video capturingdevice or an editing device. The HDR signal 500 is processed by using aninverse transform curve function 510, and a resulted signal 510 a (e.g.,linear signal) is generated and provided to a SDR generator 520 and aHDR generator 530. The SDR generator 520 processes the resulted signal510 a, to generate a SDR YUV signal 520 a. The HDR generator 530includes a new transform curve function 532, a color space transformfunction 534 and a downsampling function 536, and processes the resultedsignal 510 a, to generate a HDR YUV signal 530 a (e.g., format 4:2:0). A3D lookup table 540 is used for generating metadata 540 a according tothe SDR YUV signal 520 a and the HDR YUV signal 530 a. The metadata 540a provides additional information by describing a relation between theSDR YUV signal 520 a and the HDR YUV signal 530 a. Then, the transmitterTX3 may transmit the HDR YUV signal 530 a and the metadata 540 a in abit stream, e.g., to the receiver RX.

FIG. 6 is a transmitter TX4 according to an example of the presentinvention. The transmitter TX4 may be used for realizing the transmitterTX (the preprocessing module 110 and/or the encoder 130) in FIG. 1, andincludes modules performing different signal processing functions on aHDR YUV signal 600. A SDR generator 610 processes the HDR YUV signal600, to generate a SDR YUV signal 610 a. A 3D lookup table 620 is usedfor generating metadata 620 a according to the SDR YUV signal 610 a andthe HDR YUV signal 600. The metadata 620 a provides additionalinformation by describing a relation between the SDR YUV signal 610 aand the HDR YUV signal 600. Then, the transmitter TX4 may transmit theHDR YUV signal 600 and the metadata 620 a in a bit stream, e.g., to thereceiver RX.

FIG. 7 is a receiver RX2 according to an example of the presentinvention. The receiver RX2 may be used for realizing the receiver RX(the decoder 140 and/or the postprocessing module 160) in FIG. 1, andincludes modules performing different signal processing functions on abit stream 700. A video decoder 710 (e.g., AVS decoder) processes thebit stream 700, to generate a HDR YUV signal 710 a (e.g., format 4:2:0)and metadata 710 b. A 3D lookup table 720 is used for generating a SDRYUV signal 720 a (e.g., format 4:2:0) according to the HDR YUV signal710 a and the metadata 710 b. Then, the receiver RX2 may playback theSDR YUV signal 720 a.

The third application includes transmitting a first format of a HDRvideo sequence as a base video sequence and a second format of the HDRvideo sequence. Contrast is one of important factors in how good a TVpicture looks and it is a key part of a HDR TV. Peak brightness refersto how bright a TV can go, and is measured in a unit of “nit”. The peakbrightness reached by HDR TVs may be different. For example, a TV mayhave a peak brightness of 400 nits while another TV may have a peakbrightness of 1000 nits.

Since peak brightness of TVs may be different, it is desirable that apeak brightness of HDR content can match that of a HDR TV, to have thebest display result. To achieve this purpose, one format of a HDR videosequence is transmitted as a base video sequence and a 3D lookup tablemaps the format of the HDR video sequence to another format of the HDRvideo sequence in metadata. The supported formats of the HDR videosequence can be any of the following combinations: two HDR videosequences of different nits, hybrid Log-Gamma HDR/Perceptual QuantizerHDR (HLG HDR/PQ HDR) and PQ HDR/HLG HDR.

FIG. 8 is a transmitter TX5 according to an example of the presentinvention. The transmitter TX5 may be used for realizing the transmitterTX (the preprocessing module 110 and/or the encoder 130) in FIG. 1, andincludes modules performing different signal processing functions on rawdata 800. The raw data 800 may be generated by a video capturing deviceor an editing device. The transmitter TX5 processes the raw data 800, togenerate a HDR YUV signal 810 (e.g., format HDR HLG) and a HDR YUVsignal 820 (e.g., format HDR PQ). Formats (e.g., luminance, peakbrightness, etc.) of the HDR YUV signals 810 and 820 are different. A 3Dlookup table 830 is used for generating metadata 830 a according to theHDR YUV signals 810 and 820. The metadata 830 a provides additionalinformation by describing a relation between the HDR YUV signals 810 and820. Then, the transmitter TX5 may transmit the HDR YUV signal 820 andthe metadata 830 a in a bit stream, e.g., to the receiver RX.

FIG. 9 is a receiver RX3 according to an example of the presentinvention. The receiver RX3 may be used for realizing the receiver RX(the decoder 140 and/or the postprocessing module 160) in FIG. 1, andincludes modules performing different signal processing functions on abit stream 900. A video decoder 910 (e.g., AVS decoder) processes thebit stream 900, to generate a HDR YUV signal 910 a (e.g., format HDR PQ)and metadata 910 b. A 3D lookup table 920 is used for generating a HDRYUV signal 920 a (e.g., format HDR HLG) according to the HDR YUV signal910 a and the metadata 910 b. Then, the receiver RX3 may playback theHDR YUV signal 920 a.

If HDR raw data are retrieved from a video capturing device directly,the HDR raw data may need to be transferred to a HDR YUV 4:2:0 formatcompatible with common encoders. A procedure in HDR 10 standard isadopted in the present invention.

FIG. 10 is a HDR generator 1000 according to an example of the presentinvention. The HDR generator 1000 may be used for realizing any of theabovementioned HDR generators or for generating a HDR signal or a HDRvideo sequence, and is not limited herein. The HDR generator 1000includes modules performing different signal processing functions on aHDR source 1002. The HDR source 1002 is processed by using a curvetransform 1004, and a resulted signal 1004 a is generated and providedto a R′G′B′ to Y′CbCr conversion 1006. Accordingly, a resulted signal1006 a is generated and provided to a 10 bit quantization 1008, and aquantized signal 1008 a is generated. The quantized signal 1008 a isprocessed by a 4:4:4 to 4:2:0 conversion 1010 (i.e., downsampling), anda HDR YUV signal 1012 is generated.

Two SDR generation methods are discussed as follows. The first SDRgeneration method includes generating a SDR signal according to a HDRsignal generated by a video capturing device or an editing device. Thesecond SDR generation method includes generating a SDR signal accordingto a HDR signal which is ready for encoding/transmitting.

FIG. 11 is a SDR generator 1100 according to an example of the presentinvention. The SDR generator 1100 may be used for realizing any of theabovementioned SDR generators or for generating a SDR signal or a SDRvideo sequence, and is not limited herein. The SDR generator 1100includes modules performing different signal processing functions on alinear RGB signal 1102. The linear RGB signal 1102 is processed by usinga Luma adjustment 1104, to generate an adjusted signal 1104 a. A Lumaadjustment value Y may be computed based on R, G, B component values,and is further adapted according to the following equation:

k=Y ^((1-gamma)/gamma)  (Eq.1)

wherein gamma is a system input. After the R, G, B component values aremultiplied by the Luma adjustment value k, the R, G, B component valuesare further shaped by a SDR generation curve 1106, to generate aresulted signal 1106 a. The SDR generation curve 1106 may be selectedaccording to a ITU 1886 standard, a ITU 709 standard or a HLG standard.Then, a color space transform 1108 is performed on the resulted signal1106 a, to generate a transformed signal 1108 a. The transformed signal1108 a is processed by a 10 bit quantization 1110, to generate aquantized signal 1110 a. A downsampling 1112 is performed on thequantized signal 1110 a, to generate a SDR YUV signal 1114.

FIG. 12 is a SDR generator 1200 according to an example of the presentinvention. The SDR generator 1200 may be used for realizing any of theabovementioned SDR generators or for generating a SDR signal or a SDRvideo sequence, and is not limited herein. The SDR generator 1200includes modules performing different signal processing functions on aHDR signal 1202. The HDR signal 1202 may be a HDR YUV signal with atransform curve, which is ready for encoding/transmitting. The HDRsignal 1202 is processed by using an auto parameter calculation 1204, togenerate a mapping curve parameter f which is used for adjusting aquality of final SDR content. A luma component of the HDR signal 1202(e.g., signal Y) is processed by a HDR to SDR function 1206 (i.e., lumamapping), to generate a SDR signal 1206 a. A chroma component of the HDRsignal 1202 (e.g., signal CbCr) is processed by a downsampling function1208 and is then processed with the luma component of the HDR signal1202 by a HDR to SDR function 1210 (i.e., chroma mapping), to generate aSDR signal 1210 a. Then, a SDR YUV signal 1212 (e.g., format 4:2:0) isobtained according to the SDR signal 1206 a and the SDR signal 1210 a.That is, the luma component and the chroma component of the HDR signal1202 are mapped from HDR to SDR separately to the SDR YUV signal 1212.In one example, the chroma transform is performed in ¼ pixel resolution.One major advantage of the present example is low implementationcomplexity as no full resolution color space transform is needed.

According to the second SDR generation method shown in FIG. 12, aluminance in SDR may be generated as follows. Given a YUV 4:2:0 signalwith transform curve and Y₀, a classic mapping function may be used formapping each Luma sample to Y_(t) which is quantized to Y_(s), whereY_(s) is its SDR value ranged between 0 and (2^(N)−1). The classicmapping function may be performed according to the following equations:

$\begin{matrix}{Y_{t} = \frac{\log \left( {\frac{Y_{0}}{f} + 1} \right)}{\log \left( {\frac{P}{f} + 1} \right)}} & \left( {{Eq}.\mspace{14mu} 2} \right) \\{Y_{s} = {{\left( {2^{N} - 1} \right)Y_{t}} = {\left( {2^{N} - 1} \right) \cdot \frac{\log \left( {\frac{Y_{0}}{f} + 1} \right)}{\log \left( {\frac{P}{f} + 1} \right)}}}} & \left( {{Eq}.\mspace{14mu} 3} \right)\end{matrix}$

where P is the maximum luminance. f is a parameter for controlling acurvature of the classic mapping function and is based on an averageluma value of input HDR L_(mean). f is calculated according to thefollowing equation:

f=a*L _(mean) *L _(mean) +b*L _(mean) +c  (Eq.4)

where a=0.00001, b=0.0001, c=0.3. The values of a, b and c may bederived based on experimental results. According to the above equations,the HDR to SDR process is essentially a non-linear normalization of Y₀from [0, P] to [0, 1].

A chromaticity in SDR may be generated as follows. Given a RGB signal in¼ resolution, its linear luminance Y₀ and chroma U₀ and V₀ can becalculated as follows:

$\begin{matrix}{\left. \begin{bmatrix}R \\G \\B\end{bmatrix}\rightarrow\begin{bmatrix}Y_{0} \\U_{0} \\V_{0}\end{bmatrix} \right. = {A \cdot \begin{bmatrix}R \\G \\B\end{bmatrix}}} & \left( {{Eq}.\mspace{14mu} 5} \right)\end{matrix}$

The luminance in the HDR to SDR process is the same as that in theprevious step, except that the resolution of Y₀ and Y_(s) is ¼ of thatin the previous step. Given the ratio of Y_(s) and Y₀, the RGB signal in¼ resolution is rescaled to a RsGsBs signal according to the followingequation, where the RsGsBs signal is the SDR signal in the RGB domain.

$\begin{matrix}{\begin{bmatrix}R_{s} \\G_{s} \\B_{s}\end{bmatrix} = {\frac{Y_{s}}{Y_{0}} \cdot \begin{bmatrix}R \\G \\B\end{bmatrix}}} & \left( {{Eq}.\mspace{14mu} 6} \right)\end{matrix}$

A CbCr signal may be calculated according to the RsGsBs signal accordingto the following equation:

$\begin{matrix}{\begin{bmatrix}U_{s} \\V_{s}\end{bmatrix} = {{\begin{bmatrix}A_{2} \\A_{3}\end{bmatrix} \cdot \begin{bmatrix}R_{s} \\G_{s} \\B_{s}\end{bmatrix}} = {{\begin{bmatrix}A_{2} \\A_{3}\end{bmatrix} \cdot \frac{Y_{s}}{Y_{0}} \cdot \begin{bmatrix}R \\G \\B\end{bmatrix}} = {{\frac{Y_{s}}{Y_{0}} \cdot \begin{bmatrix}A_{2} \\A_{3}\end{bmatrix} \cdot \begin{bmatrix}R \\G \\B\end{bmatrix}} = {\frac{Y_{s}}{Y_{0}} \cdot \begin{bmatrix}U_{0} \\V_{0}\end{bmatrix}}}}}} & \left( {{Eq}.\mspace{14mu} 7} \right)\end{matrix}$

Since the objective of the HDR to SDR conversion is to reproduce thelinear light in the HDR domain to a SDR range, a non-linear function isused to map a value from a linear HDR domain to the SDR range. It maynot be necessary to converse RGB to a non-linear HDR domain first as thegoal for the HDR to SDR conversion is not conserving perceptual details.

A set of steps are proposed for preprocessing (e.g., the preprocessingmodule 110):

Step 1: Multiply Y, U, V pixel values in an original color space by k,where E′=E*k and E can be Y, U or V.

Step 2: Divide the original color space into 4×2×2 cubes.

Step 3: Collect pixel values in each cube and corresponding pixelpositions.

Step 4: Collect target color space pixel values according to the pixelpositions obtained in Step 3.

Step 5: Calculate Y, U, V color mapping coefficients in each cube basedon a 3D lookup table.

Step 6: Transmit the Y, U, V color mapping coefficients in a bit stream.

In one example, Step 5 is realized as follows. The transmitter TX onlytransmits a base video sequence and metadata. In order to reconstructadditional format(s) of the base video sequence at the receiver RX, amapping method, i.e., a 3D lookup table, between the additionalformat(s) of the base video sequence and the base video sequence isproposed.

The main idea of the 3D lookup table is using a matrix to map or topredict the additional format(s) of the base video sequence based on thebase video sequence.

Let Y_(B), U_(B), V_(B) be the base video sequence and Y_(E), U_(E),V_(E) be the additional format(s) of the base video sequence. Theirrelation can be described according to the following 3D mappingequation:

$\begin{matrix}{\begin{bmatrix}Y_{E} \\U_{E} \\V_{E}\end{bmatrix} = {{k \cdot \left( {{\begin{bmatrix}a_{y} & b_{y} & c_{y} \\a_{u} & b_{u} & c_{u} \\a_{v} & b_{v} & c_{v}\end{bmatrix} \cdot \begin{bmatrix}Y_{B} \\U_{B} \\V_{B}\end{bmatrix}} + \begin{bmatrix}d_{y} \\d_{u} \\d_{v}\end{bmatrix}} \right)} = {{\begin{bmatrix}a_{y} & b_{y} & c_{y} \\a_{u} & b_{u} & c_{u} \\a_{v} & b_{v} & c_{v}\end{bmatrix} \cdot \begin{bmatrix}{k \cdot Y_{B}} \\{k \cdot U_{B}} \\{k \cdot V_{B}}\end{bmatrix}} + {k \cdot \begin{bmatrix}d_{y} \\d_{u} \\d_{v}\end{bmatrix}}}}} & \left( {{Eq}.\mspace{14mu} 8} \right)\end{matrix}$

where k is max(Y_(E) _(_)max/Y_(B) _(_)max, 1), which is used to reducethe value range of a, b, c, d for a better transmission efficiency.Mapping coefficients can be obtained according to a least mean square(LMS) method as follows:

$\begin{matrix}{\begin{bmatrix}a_{y} & b_{y} & c_{y} & d_{y} \\a_{u} & b_{u} & c_{u} & d_{u} \\a_{v} & b_{v} & c_{v} & d_{v}\end{bmatrix} = {\arg \; \min \left\{ {{\sum\left( {Y_{E} - Y_{B}} \right)^{2}} + {\sum\left( {U_{E} - U_{B}} \right)^{2}} + {\sum\left( {V_{E} - V_{B}} \right)^{2}}} \right\}}} & \left( {{Eq}.\mspace{14mu} 9} \right)\end{matrix}$

To reduce a mapping error, a color space of the base video sequence isfurther divided into several small cubes as shown in FIG. 13, which is aschematic diagram of division of a base video color space according toan example of the present invention. For each cube, corresponding groupsof coefficients are calculated and transmitted as metadata individually.

A set of steps are proposed for postprocessing (e.g., the postprocessingmodule 160):

Step 7: Decode the Y, U, V color mapping coefficients.

Step 8: Multiply Y, U, V pixel values in the original color space by k,where E′=E*k.

Step 9: Calculate the target color space pixel values based on the 3Dlookup table.

Those skilled in the art should readily make combinations, modificationsand/or alterations on the abovementioned description and examples. Theabovementioned transmitter, receiver, description, steps, functions,modules and/or processes including suggested steps can be realized bymeans that could be hardware, software, firmware (known as a combinationof a hardware device and computer instructions and data that reside asread-only software on the hardware device), an electronic system, orcombination thereof.

Examples of the hardware may include analog circuit(s), digital circuit(s) and/or mixed circuit (s). For example, the hardware may includeapplication-specific integrated circuit(s) (ASIC(s)), field programmablegate array(s) (FPGA(s)), programmable logic device(s), coupled hardwarecomponents or combination thereof. In one example, the hardware includesgeneral-purpose processor(s), microprocessor(s), controller(s), digitalsignal processor(s) (DSP(s)) or combination thereof.

Examples of the software may include set(s) of codes, set(s) ofinstructions and/or set(s) of functions retained (e.g., stored) in astorage unit, e.g., a computer-readable medium. The computer-readablemedium may include Subscriber Identity Module (SIM), Read-Only Memory(ROM), flash memory, Random Access Memory (RAM), CD-ROM/DVD-ROM/BD-ROM,magnetic tape, hard disk, optical data storage device, non-volatilestorage unit, or combination thereof. The computer-readable medium(e.g., storage unit) may be coupled to at least one processor internally(e.g., integrated) or externally (e.g., separated). The at least oneprocessor which may include one or more modules may (e.g., be configuredto) execute the software in the computer-readable medium. The set(s) ofcodes, the set(s) of instructions and/or the set(s) of functions maycause the at least one processor, the module(s), the hardware and/or theelectronic system to perform the related steps.

To sum up, the present invention provides a device and method forhandling multiple formats of a video sequence. Thus, a transmission ofredundant video information is avoided. As a result, consumption of abandwidth and a power of a transmitter can be reduced.

Those skilled in the art will readily observe that numerousmodifications and alterations of the device and method may be made whileretaining the teachings of the invention. Accordingly, the abovedisclosure should be construed as limited only by the metes and boundsof the appended claims.

What is claimed is:
 1. A transmitter for handling multiple formats of avideo sequence, comprising: a preprocessing module, for receiving afirst format of a video sequence, to generate metadata of a secondformat of the video sequence according to the first format of the videosequence and the second format of the video sequence; and an encoder,couple to the preprocessing module, for transmitting the first format ofthe video sequence and the metadata in a bit stream to a receiver. 2.The transmitter of claim 1, wherein the first format of the videosequence is standard dynamic range (SDR), and the second format of thevideo sequence is high dynamic range (HDR).
 3. The transmitter of claim1, wherein the first format of the video sequence is HDR, and the secondformat of the video sequence is SDR.
 4. The transmitter of claim 1,wherein the first format of the video sequence is a first format of HDR,and the second format of the video sequence is a second format of HDR.5. The transmitter of claim 1, wherein the first format of the videosequence is generated by a video capturing device or an editing device.6. The transmitter of claim 1, wherein the preprocessing modulegenerates the metadata of the second format of the video sequenceaccording to the first format of the video sequence, the second formatof the video sequence and a 3D lookup table.
 7. The transmitter of claim1, wherein the metadata provides additional information by describing arelation between the first format of the video sequence and the secondformat of the video sequence.
 8. A receiver for handling multipleformats of a video sequence, comprising: a decoder, for receiving a bitstream comprising a first format of a video sequence and metadata of asecond format of the video sequence from a transmitter; and apostprocessing module, couple to the decoder, for generating the secondformat of the video sequence according to the metadata and the firstformat of the video sequence.
 9. The receiver of claim 8, wherein thefirst format of the video sequence is standard dynamic range (SDR), andthe second format of the video sequence is high dynamic range (HDR). 10.The receiver of claim 8, wherein the first format of the video sequenceis HDR, and the second format of the video sequence is SDR.
 11. Thereceiver of claim 8, wherein the first format of the video sequence is afirst format of HDR, and the second format of the video sequence is asecond format of HDR.
 12. The receiver of claim 8, wherein the firstformat of the video sequence is generated by a video capturing device oran editing device.
 13. The receiver of claim 8, wherein thepostprocessing module generates the second format of the video sequenceaccording to the metadata, the first format of the video sequence and a3D lookup table.
 14. The receiver of claim 8, wherein the metadataprovides additional information by describing a relation between thefirst format of the video sequence and the second format of the videosequence.